Vehicle rideshare localization and passenger identification for autonomous vehicles

ABSTRACT

Various disclosed embodiments include methods of locating an intended passenger of an automotive vehicle and vehicle-based systems for locating an intended passenger of an automotive vehicle. In an illustrative embodiment, a method of locating an intended passenger of an automotive vehicle includes: moving an automotive vehicle to an approximate location of an intended passenger based on a location signal from the intended passenger received via a wireless network; after moving to the approximate location of the intended passenger, receiving, by the vehicle, an image from an electronic device of the intended passenger, the image being an image of the intended passenger or an image of local physical surroundings of the intended passenger; processing the image to determine additional location information of the intended passenger based on matching the image against one or more reference images; and accepting by the vehicle, the intended passenger responsive to the image processing.

INTRODUCTION

The present disclosure relates to transportation using autonomousvehicles, and more particularly to systems and methods for pairingautonomous vehicles to their intended passengers.

The statements in this section merely provide background informationrelated to the present disclosure and may not constitute prior art.

Autonomous vehicles are of great interest for transportationapplications and can provide benefits to the ride-sharing industry.Conventionally, when an intended passenger makes a request for aride-sharing vehicle, the intended passenger may have to contact thedriver or have human interactions with the driver of the vehicle toidentify his/her particular vehicle, especially when in a crowed areasuch as a city or an airport. Such interaction, however, may bechallenging, if even possible, when the passenger is attempting tolocate an autonomous vehicle within a pool of autonomous vehicles or acrowded area with obstructions

The Society for Automotive Engineers (SAE) has defined five levels ofvehicle autonomy, ranging from level 0 (no automation) to level 5 (fullautomation), that have been adopted by the U.S. National Highway TrafficSafety Administration. For a non-autonomous vehicle (level 0), thedriver performs all driving tasks. In a vehicle with level 1-3 autonomy(driver assistance, partial automation, and conditional automationrespectively), the driver may need to control the vehicle at certaintimes, e.g., so as to stop quickly or maneuver to avoid an obstacle. Ina vehicle with level 4 autonomy (high automation), the driver's controlis optional and as such the driver may rarely interact with the vehicle.In level 5 autonomy (full automation), the driver may not drive thevehicle at all (and in this respect can be considered a passenger).

SUMMARY

Various disclosed embodiments include methods of locating an intendedpassenger of an automotive vehicle and vehicle-based systems forlocating an intended passenger of an automotive vehicle.

In an illustrative embodiment, a method of locating an intendedpassenger of an automotive vehicle includes: moving an automotivevehicle to an approximate location of an intended passenger based on alocation signal from the intended passenger received via a wirelessnetwork; after moving to the approximate location of the intendedpassenger, receiving, by the vehicle, an image from an electronic deviceof the intended passenger, the image being an image of the intendedpassenger or an image of local physical surroundings of the intendedpassenger; processing the image to determine additional locationinformation of the intended passenger based on matching the imageagainst one or more reference images; and accepting by the vehicle, theintended passenger responsive to the image processing.

In another illustrative embodiment, a vehicle-based system for locatingan intended passenger of an automotive vehicle includes: a wirelesstransceiver configured to receive a location signal via a wirelessnetwork; an image sensor for acquiring imagery proximate the automotivevehicle; a navigation system for providing guidance for moving thevehicle to an approximate location of the intended passenger based onthe location signal; and a computer processing system configured toreceive an image from an electronic device of the intended passengerafter the vehicle is moved to the approximate location of the intendedpassenger, the image being an image of the intended passenger or animage of local physical surroundings of the intended passenger, andprocessing the image to determine additional location information of theintended passenger based on matching the image against one or morereference images, the computer processing system being configured toaccept the intended passenger responsive to the image processing.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the drawings and the followingdetailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments are illustrated in referenced figures of thedrawings. It is intended that the embodiments and figures disclosedherein are to be considered illustrative rather than restrictive.

FIG. 1 is a block diagram in partial schematic form of an illustrativeframework for an approach of locating an intended passenger of anautonomous vehicle.

FIG. 2 is a flow chart of an illustrative method of locating an intendedpassenger of an autonomous vehicle.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof. In the drawings,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description, drawings, and claims are not meant to be limiting.Other embodiments may be utilized, and other changes may be made,without departing from the spirit or scope of the subject matterpresented here.

Given by way of non-limiting overview and as described in examplesherein, the present systems and methods provide systems and methods forlocating intended passengers of autonomous vehicles. For example, when apassenger requests a ride from an autonomous vehicle, e.g., using anapplication on the passenger's smartphone, that request may be relayedvia a wireless network to a backend system which then relays the requestto the vehicle via the wireless network. The request sent to the vehiclecan include a location signal, such as a global positioning system (GPS)coordinate, corresponding to the approximate location of the intendedpassenger. This location signal allows the vehicle to arrive relativelyclose to the intended passenger's actual location (e.g., within 10 feet,within 20 feet, within 50 feet, etc.). However, the vehicle may not beable to find the passenger and/or the passenger may not be able to findthe vehicle, for example, because there may be many vehicles and/or manyhumans in that approximate location. In some configurations, theintended passenger can capture a two-dimensional or three-dimensionalimage of immediately proximate streetscape structures such as buildings,landmarks, and the like, and transmit that image to the vehicle via thewireless network. The vehicle can use the image to identify and acceptthe passenger, e.g., by correlating the image to a reference imagestored in an image database whose metadata is associated with a morespecific location so as to more closely determine the intendedpassenger's actual location and then move to that location.Additionally, or alternatively, the intended passenger can display oneor more signaling image(s) from his or her smartphone, such as apredetermined color, pattern, light-flashing pattern, or sequence ofcolors and hold it up for imaging by the vehicle such that the vehiclecan recognize the predetermined signaling image(s) and accept thepassenger (e.g., from amongst a group of humans) by obtaining andprocessing the acquired signal image(s) provided by the passenger.

FIG. 1 illustrates an illustrative framework 100 for an approach oflocating an intended passenger of an autonomous vehicle according toexamples of the disclosure. The framework 100 includes a vehicle 110including an image processing and control system 120 and optionally alsoincluding an image sensor 111, an electronic device 130, and backendsystem(s) 140. The image processing and control system 120 can beconfigured so as to wirelessly communicate with the electronic device130 via the backend system(s) 140 and the wireless network, and tocontrol the vehicle 110 based thereon. The electronic device 130 caninclude a smartphone 130 from which the image processing and controlsystem 120 can receive images and/or signals indicative of the intendedpassenger's location and/or physical surroundings, and can takeappropriate actions based upon such signals and images such as describedin greater detail herein. In some configurations, the vehicle is anautonomous vehicle, such as a level 4 or level 5 autonomous vehicle.However, it should be appreciated that the present systems and methodssuitably can be used in any type of vehicle, including level 0 vehicles,level 1 vehicles, level 2 vehicles, level 3 vehicles, level 4 vehicles,and/or level 5 vehicles, as well as vehicles that use differentcategories of automation, are fully non-automated, or have any suitablelevel of automation.

An autonomous vehicle can be configured with an array of sensors,including LIDAR (light detection and ranging) and camera systems 113 foracquiring real-time video imagery of obstacles and other vehicles, GPS(global positioning system) 114, wireless communication systems forsending and receiving communications regarding road information andtraffic in real-time, as well as a computer for applying driving rulesand making driving decisions based on data and information acquired fromthe foregoing, in a suitable manner such as conventionally known in theart.

In the example illustrated in FIG. 1, the image processing and controlsystem 120 includes a processing system 121, e.g., one or more centralprocessing units (CPUs) and one or memories utilized by the processingsystem 121, such as random access memory (RAM) 122, random online memory(ROM) 123, and/or nonvolatile memory 124, as well as interfaces 125 viawhich the vehicle can communicate with the occupant, the electronicdevice 130, and/or the backend system(s) 140. Nonvolatile memory 124 caninclude one or more software modules configured to cause the processingsystem 121 to perform one or more operations such as provided herein.For example, nonvolatile memory 124 can include program instructionsconfigured to cause the processing system 121 to process one or morelocal streetscape or landscape images received from the intendedpassenger of the vehicle 110 via signals received via a wirelesstransceiver 126.

The local image(s) received from the intended passenger may then beprocessed by image processing at the processing system 121 or at aprocessing system 141 of the backend system 140 to match the image withone associated with specific location metadata. Suitable imageprocessing known in the art for this purpose will be described furtherherein.

Using that specific location derived from known image metadata, theprocessing system 121 may take one or more actions, such as guiding thevehicle 110 to the intended passenger's specific location, optionallysignaling to the intended passenger the vehicle's arrival and sendingthe intended passenger a picture of the vehicle 110, e.g., via textmessaging or via the mobile app used by the intended passenger torequest the vehicle 110. The vehicle 110 may then accept the passengerbased upon any suitable means of authentication of the passenger by thevehicle 110. For example, the vehicle processing system 121 may take animage of the person believed to be the intended passenger and then carryout image processing of that image for comparison against a known imageof the intended passenger (e.g., provided by the mobile app the intendedpassenger used to request the vehicle 110) to determine that aparticular human is the intended passenger. Suitable image processingknown in the art for this purpose will be described further herein.Alternatively, the intended passenger may be prompted using hiselectronic device to enter a predetermined code at a touchscreen ortouchpad at an exterior surface of the vehicle 110 that is recognized bythe vehicle 110.

In the configuration illustrated in FIG. 1, the image processing andcontrol system 120 includes the wireless transceiver 126 and the GPSunit 114. The wireless transceiver 126 can be configured to connect tothe electronic device 130 of the intended passenger and/or to thebackend system(s) 140 via wireless communication. For example, thewireless transceiver 126 can include a cellular transceiver configuredto provide communication between the vehicle 110 and the electronicdevice 130 of the intended passenger and/or to the backend system(s) 140via a cellular connection. The wireless transceiver 126 may also includea Bluetooth transceiver configured to permit communication with theelectronic device 130 via a Bluetooth connection, and/or a Wi-Fitransceiver configured to permit communication with the electronicdevice 130 and/or to the backend system(s) 140 via a Wi-Fi connection.The GPS unit 114 can be configured so as to obtain the GPS location ofthe vehicle 110, e.g., based on signals from respective GPS satellites.

The backend system(s) 140 can be used to control a pool of autonomousvehicles of which the vehicle 110 is a member (other vehicles of poolnot specifically illustrated). The vehicle 110 can interact with thebackend system(s) 140 hosted on one or more computer servers through awireless communication network. The computer processing system 141 ofthe backend system(s) 140 can execute software operations, programinstructions or routines to implement calculations and analyses such asprovided herein. Such program instructions, accumulated data, andprocessed data may be stored one or more non-transitorycomputer-readable memories 142 and/or one or more data stores (such asan image repository) in databases 143. Communications may be carried outaccording to a client server architecture whereby passenger electronicdevices 130 and autonomous vehicles 110 can access the remote monitoringbackend system 140 via one or more servers via one or more wirelessand/or wired communication networks.

For example, as noted above, a passenger can request an autonomousvehicle using a suitable application on his or her electronic device130, such as a smartphone. The request can include a location signal,such as the passenger's current GPS coordinate provided by a GPS module131 of the electronic device 130, and can be sent via the wirelessnetwork to the backend system(s) 140 which control the pool ofautonomous vehicles. The computer processing system 141 of the backendsystem(s) 140 is configured so as to process the request, e.g.,responsive to suitable instructions stored in memory 142, and to send asignal via the wireless network to the vehicle 110 providing thelocation signal to the vehicle for use in locating the intendedpassenger.

The wireless transceiver 126 of vehicle-based image processing andcontrol system 120 is configured to receive the location signal (e.g.,GPS coordinate) via the wireless network. A processing system such asthe processing system 121 can then process the location signal andprovide guidance control to move the vehicle 110 to that location of theintended passenger based on the location signal. For example, theprocessing system 121 may be configured to compare the current GPScoordinate of the vehicle to the GPS coordinate of the intendedpassenger and control the movement of the vehicle to the GPS coordinateof the intended passenger along a suitable route using the LIDAR andcamera systems 113, the GPS unit 114, and high definition maps as knownin the art. However, the passenger's GPS coordinate (or other locationsignal or location data) received by the vehicle 110 may not exactlycorrespond to the intended passenger's precise location when the vehicle110 arrives relatively close to the intended passenger. For example, theintended passenger may have physically moved during the time betweenwhen he or she requested the vehicle and when the vehicle arrives.Additionally, or alternatively, the GPS coordinate (or other locationsignal) received by the vehicle 110 may have been inaccurate or itsprecision may have been limited due to other factors, such as proximityto tall buildings and the like. Additionally, or alternatively, even ifthe passenger's GPS coordinate may be accurate and even if the vehiclecan reach the passenger's approximate location, the GPS coordinate alonemay not provide sufficient information for the vehicle to locate theintended passenger.

As provided herein, the use and transmission of certain images canassist the vehicle 110 in locating its intended passenger. Morespecifically, the computer processing system 121 at the vehicle 110and/or the computer processing system 141 at the backend system 140 maycan be configured to receive a local streetscape image or landscapeimage taken by the electronic device 130 at the intended passenger'sprecise location after the vehicle is moved to the approximate locationof the intended passenger, and image processing can be carried out onthat image to more precisely identify the intended passenger's locationand move the vehicle 110 to the precise location of the intendedpassenger responsive to receiving the image. For example, in someconfigurations the application with which the passenger requested thevehicle prompts the passenger to take an image of the intendedpassenger's physical surroundings. The image sensor 132 of theelectronic device 130 obtains the image and transmits the image to thetransceiver 126 of vehicle 110 via the wireless network, optionally viathe backend system(s) 140. Image processing carried out by the computerprocessing system 121 can correlate the view (within the received image)to a reference image stored in an image database for which locationmetadata is known and provides a precise location for the landscape orstreetscape view.

For example, databases 143 can include a database storing a repositoryof reference images with associated location metadata (such as GPScoordinates and/or street addresses) corresponding to such images. Thevehicle can use correlations between the reference images and imagesobtained from the intended passenger's smartphone to identify thepassenger's actual location and accept him or her. For example, thecomputer processing system 121 and/or 141, programmed with programinstructions, may be configured to submit a query, via the wirelessnetwork, to the image database 143 of the backend system(s) 140 toidentify a reference image corresponding to the view within the receivedimage. Alternatively, the image processing and control system 120 canstore such repository locally, e.g., within ROM 123 or nonvolatilememory 124, and suitably query the repository without the need to accessthe backend system(s) 140. The approximate location of the passenger,known from GPS coordinates and/or wireless communication triangulation,is communicated to the processing system 121 and/or 141. With thisapproximate location information known, only a relatively smallcollection of database images, which are tagged with geolocationmetadata, need to undergo image processing to more accurately identifythe location of the passenger. In this regard, image data of outdoorscenery whose geolocation metadata correspond to locations with apredetermined variance (radius) of the approximate location of thepassenger (e.g., from GPS coordinates of the passenger's smart phonewhen the passenger took the image) may be identified from a database(s)of image data of outdoor scenery for image processing analysis to searchfor a match between the image the passenger took and image data of theselected scenery images whose geolocation metadata falls within thepredetermined distance. Image processing techniques are known in the artfor matching one image to another image from a database of images, andany suitable technique(s) for such matching may be used to carry outthis aspect of the disclosure. For example, image processing using ScaleInvariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF),and Features from Accelerated Segment Test (FAST), which are known inthe art, may be used in this regard, such as discussed in the followingarticles, the entire contents of which are incorporated herein byreference: Lowe, “Object recognition from local scale-invariantfeatures,” Proceedings of the International Conference on ComputerVision, 1999, pp. 1150-1157; Bay et al., “Speeded-Up Robust Features(SURF),” Comput. Vis. Image Understand., vol. 110, no. 3, pp. 346-359,June 2008; Rosten et al., “Machine Learning for High-Speed CornerDetection, European Conference on Computer Vision, ECCV 2006: ComputerVision, pp. 430-443. Additional image processing methods that may beapplicable are noted in “A Survey On Image Matching Methods,” Krishnanet al., International Journal of Latest Research in Engineering andTechnology (IJLRET), January 2016, pp. 58-61, the entire contents ofwhich are incorporated herein by reference. It should be understood thatthe image data in the database(s) of image data of outdoor scenery neednot necessarily be stored as pixelated images, and such image data maybe already be vectorized or otherwise preprocessed in any suitable wayto as to facilitate efficient and speedy image processing of matchingthe image taken by the passenger to image data identified as beingwithin the location variance (radius) of the passenger's location,depending upon the image processing approach utilized.

As another example, transforming the image using a suitabletransformation, such as zero-centering the pixel data or imagenormalization so the image obtained by the passenger's smartphone issubstantially unaffected by temporal lighting conditions that differfrom the reference images in the backend system(s), and performing aquery against the database of images based on the transformed imageusing deep learning and neural networks may be used. For example, thepassenger's query image can be processed by a deep neural network thatis part of the backend system(s) which has been trained to detectlandmarks found in the database(s) of reference image data whosegeolocation metadata corresponds to locations within the predeterminedvariance of the passenger's smartphone location. Examples of suchlandmarks can be building facades, vegetation, street signs, trafficsignals, statues, etc., and combinations thereof. One or more landmarksmay be detected in the passenger's query image using the deep neuralnetwork and matched to image data of known geolocation to identify amore accurate position for the passenger. Detailed implementations ofexamples of convolutional neural network systems for identifyingmatching image data, and which may be utilized for such purpose in themethods disclosed herein, are described in the following references, theentire contents of each of which are incorporated by reference herein:Weyand et al., “PlaNeT: Photo Geolocation with Convolutional NeuralNetworks, in Leibe et al. (Eds.), Computer Vision—EECV 2016, Part VIII,Lecture Notes in Computer Science 9912, pp. 37-55 (Springer, 2016);Arandjelović et al., “NetVLAD: CNN architecture for weakly supervisedplace recognition,” 2016 IEEE Conference on Computer Vision and PatternRecognition (CVPR), Las Vegas, Nev., pp. 5297-5307 (2016); andArandjelović et al., “NetVLAD: CNN architecture for weakly supervisedplace recognition,” arXiv:1511.07247v3 [cs.CV] 2 May 2016, 17 pages(obtained from https://arxiv.org/abs/1511.07247). It will be appreciatedthat such approaches may involve training a convolutional neural network(CNN) architecture using a large reference database of outdoor sceneimages (place images) such as obtained with Google Street View TimeMachine and/or other sources. Such reference image data may associatemultiple place images for the same location wherein the multiple imagesof the same place are taken, e.g., at different times of day, fromdifferent perspectives and distances, and with different amounts ofcollateral clutter (automobiles, pedestrians, etc.). Such images may bevectorized, such as known in the art and as described in the referencedarticles above, and the CNN may be trained against known training queryimages, such as described in the articles referenced above, to enablethe CNN to match actual query images to image data of the referencedatabase(s).

Where multiple landmarks are detected in the query image taken by thepassenger, the relative location of those landmarks to one another inthe query image can be processed by the processing system 121 and/or 141to determine an accurate position of the passenger's smartphone. Forexample, the accurate geolocations of the detected/matched landmarks areknown, and the sizes (e.g., heights and/or widths, etc.) of thedetected/matched landmarks may also be known. In such case, the relativesizes and orientations of those landmarks in the passenger's query imagecan undergo further image processing by the processing system 121 and/or141 according to three-dimensional geometrical principles to identifythe originating point of the passenger's query image as indicative ofthe location of the passenger's smartphone. Estimation of cameraposition of the user's smartphone can be determined by virtue ofanalyzing by the processing system 121 and/or 141 a previously createdmap of the relevant area proximate to the smartphone's geolocation, themap being based on previously captured features from images that havebeen previously scanned and for which the location of individuallandmark features are known in three dimensions (3D). The processingsystem 121 and/or 141 may then carry out a matching analysis on theimage captured by the passenger to determine the best alignment of knownfeatures from the passenger's smartphone image to the known features ofthe previously constructed 3D map of the local area, e.g., by carryingout iterative incremental rotations of the 3D map to generateprojections that can be compared to the image acquired by the passenger.An image projection from the 3D map that minimizes differences inlocations of known landmark features relative to the image taken by thepassenger (e.g., determined by feature detection analysis imagecomparison) is correlated to a camera position within the known 3D mapto identify the location of the passenger's smartphone camera in theknown 3D environment for the picture taken by the passenger. In thisregard, scaling of the image projections generated from the known 3D mapcan be used to estimate distance of the camera position within the known3D map. The smartphone functionality which takes the picture acquired bythe passenger for this purpose can automatically be set to take thepassenger picture without any magnification to facilitate such imageanalysis to avoid any magnification artifacts that might otherwisecomplicate the analysis. Distances to known landmark features of knownsize and/or separation (e.g., known distance between two or more knownlandmark features) may also be estimated by the processing system 121and/or 141 by virtue of prior known camera calibrations for a givensmartphone camera, whereby, e.g., for images taken at no additionalmagnification, a known object width or height (e.g., a known distancebetween known landmark features) will correlate to a specific pixelseparation on the image sensor of the smartphone camera at a givendistance from the object. The detected pixel separation on the imagesensor of the smartphone camera between landmark features of knownseparation can therefore be used to estimate camera distance to theknown landmark features based on such known correlations for theparticular camera of the given smartphone. Such information can bereadily retrieved from calibration tables previously prepared forvarious cameras of various smartphones.

Thus, responsive to the query, the processing system 121 and/or 141correlates the view within the received image to reference image data,the reference image data having known geolocation information. Notably,the search of the image database need not be voluminous because theapproximate location of the intended passenger is already known asmentioned above, e.g., from the GPS location of the passenger'ssmartphone that took the query image. Thus, only reference images in thedatabase having approximate locations within a predetermined variance ofthe intended passenger's approximate location need be searched. Thisprovides for increased speed of image processing and lessens therequirements on computing resources and bandwidth.

Having identified the more precise location of the intended passengerbased on the image processing, the computer processing system 121 canexecute steps to accept the intended passenger based on the correlationbetween images. For example, based on the correlation, the processingsystem 121 can cause the vehicle 110 to move to a more precise locationnear to the intended passenger's actual location (e.g., within two feet,five feet, ten feet, 20 feet, etc., of the intended passenger's actuallocation or as close to such location as the roads may allow) and toopen the vehicle door to allow the passenger to enter the vehicle.

As another alternative, which can be used alone or in combination withone or more other options provided herein, imagery received by thevehicle 110 (e.g., via wireless communication) can include a first viewof the intended passenger himself or herself. For example, in someconfigurations the application with which the passenger requested thevehicle prompts the passenger to take an image of himself or herself.The image sensor 132 of the electronic device 130 obtains the image andtransmits the image to transceiver the 126 of the vehicle 110 via thewireless network, optionally via the backend system(s) 140. The vehicle110 can include the image sensor 111 configured to obtain a second viewof the intended passenger. For example, the processing system 121 cancontrol the image sensor 111 to obtain one or more images of theapproximate location of the intended passenger, at least one of suchimages including a view of the intended passenger. The processing system121 can then correlate the first view and the second view to identifythe intended passenger and guide the vehicle to the intended passenger'sprecise location. In this regard, positional sensors and directionalsensors associated with the vehicle's camera system and guidance systemcan determine the location of the imaged passenger based on the knownorientation and location of the vehicle and based on the knownorientation and LIDAR data associated with the image taken thatcorresponds to the image received from the intended passenger. Havingdetermined the intended passenger's precise location based on thecorrelation of images, the processing system 121 is configured tothereafter accept the passenger based on the correlation. For example,based on the correlation, the processing system 121 can cause thevehicle 110 to move to a location near to the intended passenger'sprecise location (e.g., within two feet, five feet, ten feet, 20 feet,etc., of the intended passenger's actual location or as close to suchlocation as the roads may allow) and to open the vehicle door to allowthe passenger to enter the vehicle.

As still another alternative, which can be used alone or in combinationwith one or more other options provided herein, the intended passenger'selectronic device can be used so as to generate one or more images thatthe vehicle 110 can use to locate the passenger. For example, imagerycan be displayed by a display screen 133 of the electronic device, andthe vehicle 110 can include the image sensor 111 configured to receivethe image from the electronic device by obtaining an image of thatimage. For example, the processing system 121 can transmit an imagerequest signal to the electronic device 130 via the transceiver 126 andthe wireless network (and optionally also the backend system(s) 140) todisplay predetermined imagery (e.g., the processing system 121 and/or141 may provide the predetermined imagery itself to the electronicdevice 130), and the electronic device 130 may then display the imageryresponsive to the image request signal. The imagery can include, forexample, a selected color, a selected pattern, a flashing light pattern,or a selected sequence of colors, or any other image that can be used touniquely identify the electronic device 130 as belonging to the intendedpassenger. In association with the image request, the image sensor 111may acquire the imagery, and the processing system 121 may process theimagery to confirm it is the same as the predetermined imagery, therebyprecisely locating the intended passenger.

In some configurations, responsive to the image request signal, theapplication that the passenger used to request the vehicle can promptthe intended passenger to hold up the electronic device 130 in such amanner that the imagery displayed on the display screen 133 of thedevice can be imaged by the image sensor 111 of the vehicle 110. Imageprocessing carried out by the processing system 121 compares the imagethat the image sensor 111 obtained of the image displayed on the displayscreen 133 to the information about the image that was obtained inassociation with the image request. In response to a match between theimage and the information, the processing system 121 makes adetermination to accept the passenger.

In any such approaches, the vehicle may then accept the passenger basedupon any suitable means of authentication of the passenger by thevehicle 110. For example, the vehicle processing system 121 may take animage of the person believed to be the intended passenger and then carryout image processing of that image for comparison against a known imageof the intended passenger (e.g., provided by the mobile app the intendedpassenger used to request the vehicle 110) to determine that aparticular human is the intended passenger. Suitable image processingknown in the art for this purpose was mentioned above such as, e.g.,Scale Invariant Feature Transform (SIFT), Speeded-Up Robust Features(SURF), and Features from Accelerated Segment Test (FAST).

Alternatively, the intended passenger may be prompted using his or herelectronic device 130 to enter a predetermined code at a touchscreen ortouchpad at an exterior surface of the vehicle 110 that the vehiclerecognizes for authentication. Upon authentication, the processingsystem 121 and/or 141 may unlock the vehicle 110 permit access for theintended passenger.

FIG. 2 illustrates a flow chart of an illustrative method 200 oflocating an intended passenger of an autonomous vehicle, according toexamples of the disclosure. It should be appreciated that althoughoperations of the method 200 as illustrated in FIG. 2 are described withreference to the framework 100 illustrated in FIG. 1, such operationssuitably can be used with any appropriate combination of hardware andsoftware, e.g., any non-autonomous or autonomous vehicle.

The method 200 illustrated in FIG. 2 includes at a block 202 moving, bythe vehicle, to an approximate location of the intended passenger basedon a location signal received via a wireless network. For example,responsive to a vehicle request that includes the location signal andmay be generated at the electronic device 130 and processed through thewireless network and the optional backend system(s) 140 such asillustrated in FIG. 1, vehicle 110 can move to an approximate locationof the intended passenger, such as described previously herein.

Referring again to FIG. 2, the method 200 can include at a block 204,after moving to the approximate location of the intended passenger,receiving, by the vehicle, an image from an electronic device of theintended passenger wherein the image is indicative of the local physicalsurroundings, e.g., local streetscape or landscape of the intendedpassenger. As described in greater detail with reference to FIG. 1, thevehicle 110 can receive any suitable combination of images from theelectronic device 130 via any suitable modes of communication. Forexample, the image can be obtained by the imaging sensor 132 of theelectronic device 130 and received by the vehicle 110 via the wirelessnetwork. The image can include a view of the intended passenger'sphysical surroundings or a view of the intended passenger that thevehicle receives via the wireless network, and/or can be displayed bythe electronic device 130 and received by the vehicle via the imagesensor 111. Any suitable number and types of such images suitably can beused in combination with one another.

The method 200 may also include at a block 206, processing the image todetermine additional location information of the intended passenger,e.g., a more precise location of the intended passenger, based on imagematching the image against an image database containing imagesassociated with location metadata. Illustrative image processingapproaches for carrying out the image matching were noted previouslyherein.

Referring to FIG. 2, the method 200 can include at a block 208accepting, by the vehicle, the intended passenger responsive toreceiving the image (206). For example, in some configurations thevehicle 110 accepts the passenger based on a correlation between theview of the intended passenger's physical surroundings and a locationcorresponding to a reference image stored in an image database and/orbased on a correlation between the view of the intended passenger andview of the intended passenger obtained using the image sensor 111 ofthe vehicle. Additionally, or alternatively, the image can include aselected color, a selected pattern, selected flashing light pattern, ora selected sequence of colors that is broadcast by the display device133 of the electronic device 130, and the vehicle 110 accepts thepassenger based on a match between the broadcast image and informationthat the vehicle 110 has regarding the image that the electronic device130 is expected to display, such as described previously herein. Thevehicle's acceptance of the passenger further can involve other exchangeof information by a variety of means. For example, passenger acceptancemay include exchange of an unlock code sent by the vehicle to theintended passenger by text message or popup message on the smartphoneapp that provide the functionality described herein, which the user mayaccept by any suitable finger swipe or press in order to open the doorof the vehicle. Acceptance of the passenger may involve an audiblemessage spoken to the intended passenger by the automated voicetechnology of the vehicle by which the user may confirm that the vehicleis in fact the proper vehicle destined for that intended passenger. Anyother suitable means of exchanging information to confirm and accept thepassenger may also be used.

This written description describes illustrative embodiments, but othervariations fall within scope of the disclosure. For example, the systemsand methods may include and utilize data signals conveyed via networks(e.g., local area network, wide area network, internet, combinationsthereof, etc.), fiber optic medium, carrier waves, wireless networks,etc. for communication with one or more data processing devices. Thedata signals can carry any or all of the data disclosed herein that isprovided to or from a device.

The methods and systems described herein may be implemented on manydifferent types of processing devices by program code comprising programinstructions that are executable by the device processing system. Thesoftware program instructions may include source code, object code,machine code, or any other stored data that is operable to cause aprocessing system to perform the methods and operations describedherein. Any suitable computer languages may be used such as C, C++,Java, etc., as will be appreciated by those skilled in the art. Otherimplementations may also be used, however, such as firmware or evenappropriately designed hardware configured to carry out the methods andsystems described herein.

In various embodiments, a non-transitory computer readable medium forcarrying out the illustrative approach may include program instructionsadapted to cause a computer processing system to execute theabove-mentioned steps. The systems' and methods' data (e.g.,associations, mappings, data input, data output, intermediate dataresults, final data results, etc.) may be stored and implemented in oneor more different types of computer-implemented data stores, such asdifferent types of storage devices and programming constructs (e.g.,RAM, ROM, Flash memory, flat files, databases, programming datastructures, programming variables, IF-THEN (or similar type) statementconstructs, etc.). It is noted that data structures describe formats foruse in organizing and storing data in databases, programs, memory, orother non-transitory computer-readable media for use by a computerprogram.

The computer components, software modules, functions, data stores anddata structures described herein may be connected directly or indirectlyto each other in order to allow the flow of data needed for theiroperations. It is also noted that a module or processor includes but isnot limited to a unit of code that performs a software operation, andcan be implemented for example as a subroutine unit of code, or as asoftware function unit of code, or as an object (as in anobject-oriented paradigm), or as an applet, or in a computer scriptlanguage, or as another type of computer code. The software componentsand/or functionality may be located on a single computer or distributedacross multiple computers depending upon the situation at hand.

It should be understood that as used in the description herein andthroughout the claims that follow, the meaning of “a,” “an,” and “the”includes plural reference unless the context clearly dictates otherwise.Also, as used in the description herein and throughout the claims thatfollow, the meaning of “in” includes “in” and “on” unless the contextclearly dictates otherwise. Finally, as used in the description hereinand throughout the claims that follow, the meanings of “and” and “or”include both the conjunctive and disjunctive and may be usedinterchangeably unless the context expressly dictates otherwise; thephrase “exclusive or” may be used to indicate situation where only thedisjunctive meaning may apply. In addition, as used in the descriptionherein and throughout the claims that follow, the meaning of “about”and/or “approximately” refers to ±10% of the quantity indicated, unlessotherwise indicated.

While the present disclosure has been described in terms of illustrativeembodiments, it will be understood by those skilled in the art thatvarious modifications can be made thereto without departing from thescope of the claimed subject matter as set forth in the claims.

What is claimed is:
 1. A method of locating an intended passenger of anautomotive vehicle, the method comprising: causing, by a processor, thevehicle to move to an approximate location of an intended passengerbased on a location signal from the intended passenger received via awireless network; prompting, by the processor, the intended passenger toobtain an image using an electronic device after the vehicle reaches theapproximate location of the intended passenger, the image being an imageof local physical surroundings of the intended passenger; receiving, bythe processor, the image from the electronic device of the intendedpassenger; processing, by the processor, the image to determineadditional location information of the intended passenger based onmatching the image against one or more reference images selected basedon a proximity of the one or more reference images to the locationsignal from the intended passenger; and based on the determinedadditional location information, causing, by the processor, the vehicleto move to a more accurate location of the intended passenger.
 2. Themethod of claim 1, wherein the image is obtained by an imaging sensor ofthe electronic device and is received by the vehicle via the wirelessnetwork.
 3. The method of claim 1, wherein the image includes a view ofthe local streetscape or local landscape at the intended passenger'slocation.
 4. The method of claim 1, further comprising accepting, by theprocessor, the intended passenger responsive to the image processing,wherein the processor accepts the passenger based on a correlationbetween the image from the intended passenger and a reference imageassociated with location metadata stored in an image database.
 5. Themethod of claim 1, wherein the image includes a first view of theintended passenger.
 6. The method of claim 5, further comprising, by theprocessor, obtaining a second view of the intended passenger using animage sensor of the vehicle, wherein the processor accepts the passengerbased on a correlation between the first view and the second view. 7.The method of claim 1, wherein: an identifying image is displayed by adisplay screen of the electronic device; and the vehicle receives theidentifying image by using an image sensor of the vehicle to obtain animage of that identifying image.
 8. The method of claim 7, wherein: thevehicle transmits an image request signal to the electronic device viathe wireless network; and the electronic device generates theidentifying image responsive to the image request signal.
 9. The methodof claim 7, wherein the identifying image includes a selected color, aselected pattern, a selected light-flash pattern, or a selected sequenceof colors.
 10. The method of claim 1, wherein the location signalincludes a global positioning system (GPS) coordinate of the intendedpassenger.
 11. A vehicle-based system for locating an intended passengerof an automotive vehicle, the system comprising: a wireless transceiverconfigured to receive a location signal via a wireless network; an imagesensor for acquiring imagery proximate the vehicle; a navigation systemfor providing guidance for moving the vehicle to an approximate locationof the intended passenger based on the location signal; and a computerprocessing system configured to prompt the intended passenger to obtainan image using an electronic device after the vehicle reaches theapproximate location of the intended passenger, the image being an imageof local physical surroundings of the intended passenger, receive theimage from the electronic device of the intended passenger, processingthe image to determine additional location information of the intendedpassenger based on matching the image against one or more referenceimages selected based on a proximity of the one or more reference imagesto the location signal, and moving the vehicle to a more accuratelocation of the intended passenger based on the additional locationinformation.
 12. The system of claim 11, wherein the image is obtainedby an imaging sensor of the electronic device and is received by thetransceiver via the wireless network.
 13. The system of claim 11,wherein the image includes a view of the intended passenger's localstreetscape or local landscape.
 14. The system of claim 11, wherein: thecomputer processing system is configured to correlate the view and alocation corresponding to a reference image stored in an image database;and the computer processing system is configured to accept the passengerbased on the correlation.
 15. The system of claim 11, wherein the imageincludes a first view of the intended passenger.
 16. The system of claim15, wherein: the image sensor is configured to obtain a second view ofthe intended passenger; the processing system is configured to correlatethe first view and the second view; and the processing system isconfigured to accept the passenger based on the correlation.
 17. Thesystem of claim 11, wherein: an identifying image is displayed by adisplay screen of the electronic device; and the image sensor isconfigured to acquire an image of the identifying image displayed on thedisplay screen of the electronic device.
 18. The system of claim 17,wherein: the processing system is configured to cause transmission of animage request signal to the electronic device via the transceiver andthe wireless network; and the electronic device generates theidentifying image responsive to the image request signal.
 19. The systemof claim 17, wherein the identifying image includes a selected color, aselected pattern, a selected light-flash pattern, or a selected sequenceof colors.
 20. The system of claim 11, wherein the location signalincludes a global positioning system (GPS) coordinate of the intendedpassenger.